{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Project 4: Multi-factor Model\n",
    "## Instructions\n",
    "Each problem consists of a function to implement and instructions on how to implement the function.  The parts of the function that need to be implemented are marked with a `# TODO` comment. After implementing the function, run the cell to test it against the unit tests we've provided. For each problem, we provide one or more unit tests from our `project_tests` package. These unit tests won't tell you if your answer is correct, but will warn you of any major errors. Your code will be checked for the correct solution when you submit it to Udacity.\n",
    "\n",
    "## Packages\n",
    "When you implement the functions, you'll only need to you use the packages you've used in the classroom, like [Pandas](https://pandas.pydata.org/) and [Numpy](http://www.numpy.org/). These packages will be imported for you. We recommend you don't add any import statements, otherwise the grader might not be able to run your code.\n",
    "\n",
    "The other packages that we're importing are `project_helper` and `project_tests`. These are custom packages built to help you solve the problems.  The `project_helper` module contains utility functions and graph functions. The `project_tests` contains the unit tests for all the problems.\n",
    "\n",
    "### Install Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "!{sys.executable} -m pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cvxpy as cvx\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import time\n",
    "import project_tests\n",
    "import project_helper\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "plt.style.use('ggplot')\n",
    "plt.rcParams['figure.figsize'] = (14, 8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Data Bundle\n",
    "We'll be using Zipline to handle our data. We've created a end of day data bundle for this project. Run the cell below to register this data bundle in zipline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import project_helper\n",
    "from zipline.data import bundles\n",
    "\n",
    "os.environ['ZIPLINE_ROOT'] = os.path.join(os.getcwd(), '..', '..', 'data', 'project_4_eod')\n",
    "\n",
    "ingest_func = bundles.csvdir.csvdir_equities(['daily'], project_helper.EOD_BUNDLE_NAME)\n",
    "bundles.register(project_helper.EOD_BUNDLE_NAME, ingest_func)\n",
    "\n",
    "print('Data Registered')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build Pipeline Engine\n",
    "We'll be using Zipline's pipeline package to access our data for this project. To use it, we must build a pipeline engine. Run the cell below to build the engine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from zipline.pipeline import Pipeline\n",
    "from zipline.pipeline.factors import AverageDollarVolume\n",
    "from zipline.utils.calendars import get_calendar\n",
    "\n",
    "\n",
    "universe = AverageDollarVolume(window_length=120).top(500) \n",
    "trading_calendar = get_calendar('NYSE') \n",
    "bundle_data = bundles.load(project_helper.EOD_BUNDLE_NAME)\n",
    "engine = project_helper.build_pipeline_engine(bundle_data, trading_calendar)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View Data\n",
    "With the pipeline engine built, let's get the stocks at the end of the period in the universe we're using. We'll use these tickers to generate the returns data for the our risk model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "universe_end_date = pd.Timestamp('2016-01-05', tz='UTC')\n",
    "\n",
    "universe_tickers = engine\\\n",
    "    .run_pipeline(\n",
    "        Pipeline(screen=universe),\n",
    "        universe_end_date,\n",
    "        universe_end_date)\\\n",
    "    .index.get_level_values(1)\\\n",
    "    .values.tolist()\n",
    "    \n",
    "universe_tickers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get Returns\n",
    "Not that we have our pipeline built, let's access the returns data. We'll start by building a data portal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from zipline.data.data_portal import DataPortal\n",
    "\n",
    "\n",
    "data_portal = DataPortal(\n",
    "    bundle_data.asset_finder,\n",
    "    trading_calendar=trading_calendar,\n",
    "    first_trading_day=bundle_data.equity_daily_bar_reader.first_trading_day,\n",
    "    equity_minute_reader=None,\n",
    "    equity_daily_reader=bundle_data.equity_daily_bar_reader,\n",
    "    adjustment_reader=bundle_data.adjustment_reader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make the code easier to read, we've built the helper function `get_pricing` to get the pricing from the data portal. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_pricing(data_portal, trading_calendar, assets, start_date, end_date, field='close'):\n",
    "    end_dt = pd.Timestamp(end_date.strftime('%Y-%m-%d'), tz='UTC', offset='C')\n",
    "    start_dt = pd.Timestamp(start_date.strftime('%Y-%m-%d'), tz='UTC', offset='C')\n",
    "\n",
    "    end_loc = trading_calendar.closes.index.get_loc(end_dt)\n",
    "    start_loc = trading_calendar.closes.index.get_loc(start_dt)\n",
    "\n",
    "    return data_portal.get_history_window(\n",
    "        assets=assets,\n",
    "        end_dt=end_dt,\n",
    "        bar_count=end_loc - start_loc,\n",
    "        frequency='1d',\n",
    "        field=field,\n",
    "        data_frequency='daily')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View Data\n",
    "Let's get returns data for our risk model using the `get_pricing` function. For this model, we'll be looking back to 5 years of data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "five_year_returns = \\\n",
    "    get_pricing(\n",
    "        data_portal,\n",
    "        trading_calendar,\n",
    "        universe_tickers,\n",
    "        universe_end_date - pd.DateOffset(years=5),\n",
    "        universe_end_date)\\\n",
    "    .pct_change()[1:].fillna(0)\n",
    "\n",
    "five_year_returns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Statistical Risk Model\n",
    "It's time to build the risk model. You'll be creating a statistical risk model using PCA. So, the first thing is building the PCA model.\n",
    "## Fit PCA\n",
    "Implement `fit_pca` to fit a PCA model to the returns data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "\n",
    "\n",
    "def fit_pca(returns, num_factor_exposures, svd_solver):\n",
    "    \"\"\"\n",
    "    Fit PCA model with returns.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    returns : DataFrame\n",
    "        Returns for each ticker and date\n",
    "    num_factor_exposures : int\n",
    "        Number of factors for PCA\n",
    "    svd_solver: str\n",
    "        The solver to use for the PCA model\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    pca : PCA\n",
    "        Model fit to returns\n",
    "    \"\"\"\n",
    "    #TODO: Implement function\n",
    "    \n",
    "    return None\n",
    "\n",
    "\n",
    "project_tests.test_fit_pca(fit_pca)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View Data\n",
    "Let's see what the model looks like. First, we'll look at the PCA components."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_factor_exposures = 20\n",
    "pca = fit_pca(five_year_returns, num_factor_exposures, 'full')\n",
    "\n",
    "pca.components_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also look at the PCA's percent of variance explained by each factor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.bar(np.arange(num_factor_exposures), pca.explained_variance_ratio_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will see that the first factor dominates. The precise definition of each factor in a latent model is unknown, however we can guess at the likely interpretation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Factor Betas\n",
    "Implement `factor_betas` to get the factor betas from the PCA model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def factor_betas(pca, factor_beta_indices, factor_beta_columns):\n",
    "    \"\"\"\n",
    "    Get the factor betas from the PCA model.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    pca : PCA\n",
    "        Model fit to returns\n",
    "    factor_beta_indices : 1 dimensional Ndarray\n",
    "        Factor beta indices\n",
    "    factor_beta_columns : 1 dimensional Ndarray\n",
    "        Factor beta columns\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    factor_betas : DataFrame\n",
    "        Factor betas\n",
    "    \"\"\"\n",
    "    assert len(factor_beta_indices.shape) == 1\n",
    "    assert len(factor_beta_columns.shape) == 1\n",
    "    \n",
    "    #TODO: Implement function\n",
    "    \n",
    "    return None\n",
    "\n",
    "\n",
    "project_tests.test_factor_betas(factor_betas)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View Data\n",
    "Let's view the factor betas from this model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "risk_model = {}\n",
    "risk_model['factor_betas'] = factor_betas(pca, five_year_returns.columns.values, np.arange(num_factor_exposures))\n",
    "\n",
    "risk_model['factor_betas']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Factor Returns\n",
    "Implement `factor_returns` to get the factor returns from the PCA model using the returns data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def factor_returns(pca, returns, factor_return_indices, factor_return_columns):\n",
    "    \"\"\"\n",
    "    Get the factor returns from the PCA model.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    pca : PCA\n",
    "        Model fit to returns\n",
    "    returns : DataFrame\n",
    "        Returns for each ticker and date\n",
    "    factor_return_indices : 1 dimensional Ndarray\n",
    "        Factor return indices\n",
    "    factor_return_columns : 1 dimensional Ndarray\n",
    "        Factor return columns\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    factor_returns : DataFrame\n",
    "        Factor returns\n",
    "    \"\"\"\n",
    "    assert len(factor_return_indices.shape) == 1\n",
    "    assert len(factor_return_columns.shape) == 1\n",
    "    \n",
    "    #TODO: Implement function\n",
    "    \n",
    "    return None\n",
    "\n",
    "\n",
    "project_tests.test_factor_returns(factor_returns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View Data\n",
    "Let's see what these factor returns looks like over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "risk_model['factor_returns'] = factor_returns(\n",
    "    pca,\n",
    "    five_year_returns,\n",
    "    five_year_returns.index,\n",
    "    np.arange(num_factor_exposures))\n",
    "\n",
    "risk_model['factor_returns'].cumsum().plot(legend=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Factor Covariance Matrix\n",
    "Implement `factor_cov_matrix` to get the factor covariance matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def factor_cov_matrix(factor_returns, ann_factor):\n",
    "    \"\"\"\n",
    "    Get the factor covariance matrix\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    factor_returns : DataFrame\n",
    "        Factor returns\n",
    "    ann_factor : int\n",
    "        Annualization factor\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    factor_cov_matrix : 2 dimensional Ndarray\n",
    "        Factor covariance matrix\n",
    "    \"\"\"\n",
    "    \n",
    "    #TODO: Implement function\n",
    "    \n",
    "    return None\n",
    "\n",
    "\n",
    "project_tests.test_factor_cov_matrix(factor_cov_matrix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ann_factor = 252\n",
    "risk_model['factor_cov_matrix'] = factor_cov_matrix(risk_model['factor_returns'], ann_factor)\n",
    "\n",
    "risk_model['factor_cov_matrix']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Idiosyncratic Variance Matrix\n",
    "Implement `idiosyncratic_var_matrix` to get the idiosyncratic variance matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def idiosyncratic_var_matrix(returns, factor_returns, factor_betas, ann_factor):\n",
    "    \"\"\"\n",
    "    Get the idiosyncratic variance matrix\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    returns : DataFrame\n",
    "        Returns for each ticker and date\n",
    "    factor_returns : DataFrame\n",
    "        Factor returns\n",
    "    factor_betas : DataFrame\n",
    "        Factor betas\n",
    "    ann_factor : int\n",
    "        Annualization factor\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    idiosyncratic_var_matrix : DataFrame\n",
    "        Idiosyncratic variance matrix\n",
    "    \"\"\"\n",
    "    \n",
    "    #TODO: Implement function\n",
    "    \n",
    "    return None\n",
    "\n",
    "\n",
    "project_tests.test_idiosyncratic_var_matrix(idiosyncratic_var_matrix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "risk_model['idiosyncratic_var_matrix'] = idiosyncratic_var_matrix(five_year_returns, risk_model['factor_returns'], risk_model['factor_betas'], ann_factor)\n",
    "\n",
    "risk_model['idiosyncratic_var_matrix']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Idiosyncratic Variance Vector\n",
    "Implement `idiosyncratic_var_vector` to get the idiosyncratic variance Vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def idiosyncratic_var_vector(returns, idiosyncratic_var_matrix):\n",
    "    \"\"\"\n",
    "    Get the idiosyncratic variance vector\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    returns : DataFrame\n",
    "        Returns for each ticker and date\n",
    "    idiosyncratic_var_matrix : DataFrame\n",
    "        Idiosyncratic variance matrix\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    idiosyncratic_var_vector : DataFrame\n",
    "        Idiosyncratic variance Vector\n",
    "    \"\"\"\n",
    "    \n",
    "    #TODO: Implement function\n",
    "    \n",
    "    return None\n",
    "\n",
    "\n",
    "project_tests.test_idiosyncratic_var_vector(idiosyncratic_var_vector)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "risk_model['idiosyncratic_var_vector'] = idiosyncratic_var_vector(five_year_returns, risk_model['idiosyncratic_var_matrix'])\n",
    "\n",
    "risk_model['idiosyncratic_var_vector']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predict using the Risk Model\n",
    "Using the data we calculated in the risk model, implement `predict_portfolio_risk` to predict the portfolio risk using the formula $ \\sqrt{X^{T}(BFB^{T} + S)X} $ where:\n",
    "- $ X $ is the portfolio weights\n",
    "- $ B $ is the factor betas\n",
    "- $ F $ is the factor covariance matrix\n",
    "- $ S $ is the idiosyncratic variance matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_portfolio_risk(factor_betas, factor_cov_matrix, idiosyncratic_var_matrix, weights):\n",
    "    \"\"\"\n",
    "    Get the predicted portfolio risk\n",
    "    \n",
    "    Formula for predicted portfolio risk is sqrt(X.T(BFB.T + S)X) where:\n",
    "      X is the portfolio weights\n",
    "      B is the factor betas\n",
    "      F is the factor covariance matrix\n",
    "      S is the idiosyncratic variance matrix\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    factor_betas : DataFrame\n",
    "        Factor betas\n",
    "    factor_cov_matrix : 2 dimensional Ndarray\n",
    "        Factor covariance matrix\n",
    "    idiosyncratic_var_matrix : DataFrame\n",
    "        Idiosyncratic variance matrix\n",
    "    weights : DataFrame\n",
    "        Portfolio weights\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    predicted_portfolio_risk : float\n",
    "        Predicted portfolio risk\n",
    "    \"\"\"\n",
    "    assert len(factor_cov_matrix.shape) == 2\n",
    "    \n",
    "    #TODO: Implement function\n",
    "    \n",
    "    return None\n",
    "\n",
    "\n",
    "project_tests.test_predict_portfolio_risk(predict_portfolio_risk)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View Data\n",
    "Let's see what the portfolio risk would be if we had even weights across all stocks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_weights = pd.DataFrame(np.repeat(1/len(universe_tickers), len(universe_tickers)), universe_tickers)\n",
    "\n",
    "predict_portfolio_risk(\n",
    "    risk_model['factor_betas'],\n",
    "    risk_model['factor_cov_matrix'],\n",
    "    risk_model['idiosyncratic_var_matrix'],\n",
    "    all_weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create Alpha Factors\n",
    "With the profile risk calculated, it's time to start working on the alpha factors. In this project, we'll create the following factors:\n",
    "- Momentum 1 Year Factor\n",
    "- Mean Reversion 5 Day Sector Neutral Factor\n",
    "- Mean Reversion 5 Day Sector Neutral Smoothed Factor\n",
    "- Overnight Sentiment Factor\n",
    "- Overnight Sentiment Smoothed Factor\n",
    "\n",
    "## Momentum 1 Year Factor\n",
    "Each factor will have a hypothesis that goes with it. For this factor, it is \"Higher past 12-month (252 days) returns are proportional to future return.\" Using that hypothesis, we've generated this code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from zipline.pipeline.factors import Returns\n",
    "\n",
    "def momentum_1yr(window_length, universe, sector):\n",
    "    return Returns(window_length=window_length, mask=universe) \\\n",
    "        .demean(groupby=sector) \\\n",
    "        .rank() \\\n",
    "        .zscore()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mean Reversion 5 Day Sector Neutral Factor\n",
    "Now it's time for you to implement `mean_reversion_5day_sector_neutral` using the hypothesis \"Short-term outperformers(underperformers) compared to their sector will revert.\" Use the returns data from `universe`, demean using the sector data to partition, rank, then converted to a zscore."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mean_reversion_5day_sector_neutral(window_length, universe, sector):\n",
    "    \"\"\"\n",
    "    Generate the mean reversion 5 day sector neutral factor\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    window_length : int\n",
    "        Returns window length\n",
    "    universe : Zipline Filter\n",
    "        Universe of stocks filter\n",
    "    sector : Zipline Classifier\n",
    "        Sector classifier\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    factor : Zipline Factor\n",
    "        Mean reversion 5 day sector neutral factor\n",
    "    \"\"\"\n",
    "    \n",
    "    #TODO: Implement function\n",
    "    \n",
    "    return None\n",
    "\n",
    "\n",
    "project_tests.test_mean_reversion_5day_sector_neutral(mean_reversion_5day_sector_neutral)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View Data\n",
    "Let's see what some of the factor data looks like. For calculating factors, we'll be looking back 2 years.\n",
    "\n",
    "**Note:** _Going back 2 years falls on a day when the market is closed. Pipeline package doesn't handle start or end dates that don't fall on days when the market is open. To fix this, we went back 2 extra days to fall on the next day when the market is open._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "factor_start_date = universe_end_date - pd.DateOffset(years=2, days=2)\n",
    "sector = project_helper.Sector()\n",
    "window_length = 5\n",
    "\n",
    "pipeline = Pipeline(screen=universe)\n",
    "pipeline.add(\n",
    "    mean_reversion_5day_sector_neutral(window_length, universe, sector),\n",
    "    'Mean_Reversion_5Day_Sector_Neutral')\n",
    "engine.run_pipeline(pipeline, factor_start_date, universe_end_date)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mean Reversion 5 Day Sector Neutral Smoothed Factor\n",
    "Taking the output of the previous factor, let's create a smoothed version. Implement `mean_reversion_5day_sector_neutral_smoothed` to generate a mean reversion 5 fay sector neutral smoothed factor. Call the `mean_reversion_5day_sector_neutral` function to get the unsmoothed factor, then use `SimpleMovingAverage` function to smooth it. You'll have to apply rank and zscore again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from zipline.pipeline.factors import SimpleMovingAverage\n",
    "\n",
    "def mean_reversion_5day_sector_neutral_smoothed(window_length, universe, sector):\n",
    "    \"\"\"\n",
    "    Generate the mean reversion 5 day sector neutral smoothed factor\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    window_length : int\n",
    "        Returns window length\n",
    "    universe : Zipline Filter\n",
    "        Universe of stocks filter\n",
    "    sector : Zipline Classifier\n",
    "        Sector classifier\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    factor : Zipline Factor\n",
    "        Mean reversion 5 day sector neutral smoothed factor\n",
    "    \"\"\"\n",
    "    \n",
    "    #TODO: Implement function\n",
    "    \n",
    "    return None\n",
    "\n",
    "\n",
    "project_tests.test_mean_reversion_5day_sector_neutral_smoothed(mean_reversion_5day_sector_neutral_smoothed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View Data\n",
    "Let's see what some of the smoothed data looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = Pipeline(screen=universe)\n",
    "pipeline.add(\n",
    "    mean_reversion_5day_sector_neutral_smoothed(5, universe, sector),\n",
    "    'Mean_Reversion_5Day_Sector_Neutral_Smoothed')\n",
    "engine.run_pipeline(pipeline, factor_start_date, universe_end_date)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overnight Sentiment Factor\n",
    "For this factor, were using the hypothesis from the paper [_Overnight Returns and Firm-Specific Investor Sentiment_](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2554010)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from zipline.pipeline.data import USEquityPricing\n",
    "\n",
    "\n",
    "class CTO(Returns):\n",
    "    \"\"\"\n",
    "    Computes the overnight return, per hypothesis from\n",
    "    https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2554010\n",
    "    \"\"\"\n",
    "    inputs = [USEquityPricing.open, USEquityPricing.close]\n",
    "    \n",
    "    def compute(self, today, assets, out, opens, closes):\n",
    "        \"\"\"\n",
    "        The opens and closes matrix is 2 rows x N assets, with the most recent at the bottom.\n",
    "        As such, opens[-1] is the most recent open, and closes[0] is the earlier close\n",
    "        \"\"\"\n",
    "        out[:] = (opens[-1] - closes[0]) / closes[0]\n",
    "\n",
    "        \n",
    "class TrailingOvernightReturns(Returns):\n",
    "    \"\"\"\n",
    "    Sum of trailing 1m O/N returns\n",
    "    \"\"\"\n",
    "    window_safe = True\n",
    "    \n",
    "    def compute(self, today, asset_ids, out, cto):\n",
    "        out[:] = np.nansum(cto, axis=0)\n",
    "\n",
    "        \n",
    "def overnight_sentiment(cto_window_length, trail_overnight_returns_window_length, universe):\n",
    "    cto_out = CTO(mask=universe, window_length=cto_window_length)\n",
    "    return TrailingOvernightReturns(inputs=[cto_out], window_length=trail_overnight_returns_window_length) \\\n",
    "        .rank() \\\n",
    "        .zscore()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overnight Sentiment Smoothed Factor\n",
    "Just like the factor you implemented, we'll also smooth this factor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def overnight_sentiment_smoothed(cto_window_length, trail_overnight_returns_window_length, universe):\n",
    "    unsmoothed_factor = overnight_sentiment(cto_window_length, trail_overnight_returns_window_length, universe)\n",
    "    return SimpleMovingAverage(inputs=[unsmoothed_factor], window_length=trail_overnight_returns_window_length) \\\n",
    "        .rank() \\\n",
    "        .zscore()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Combine the Factors to a single Pipeline\n",
    "With all the factor implementations done, let's add them to a pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "universe = AverageDollarVolume(window_length=120).top(500)\n",
    "sector = project_helper.Sector()\n",
    "\n",
    "pipeline = Pipeline(screen=universe)\n",
    "pipeline.add(\n",
    "    momentum_1yr(252, universe, sector),\n",
    "    'Momentum_1YR')\n",
    "pipeline.add(\n",
    "    mean_reversion_5day_sector_neutral(5, universe, sector),\n",
    "    'Mean_Reversion_5Day_Sector_Neutral')\n",
    "pipeline.add(\n",
    "    mean_reversion_5day_sector_neutral_smoothed(5, universe, sector),\n",
    "    'Mean_Reversion_5Day_Sector_Neutral_Smoothed')\n",
    "pipeline.add(\n",
    "    overnight_sentiment(2, 5, universe),\n",
    "    'Overnight_Sentiment')\n",
    "pipeline.add(\n",
    "    overnight_sentiment_smoothed(2, 5, universe),\n",
    "    'Overnight_Sentiment_Smoothed')\n",
    "all_factors = engine.run_pipeline(pipeline, factor_start_date, universe_end_date)\n",
    "\n",
    "all_factors.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluate Alpha Factors\n",
    "*Note:* _We're evaluating the alpha factors using delay of 1_\n",
    "## Get Pricing Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import alphalens as al\n",
    "\n",
    "assets = all_factors.index.levels[1].values.tolist()\n",
    "pricing = get_pricing(\n",
    "    data_portal,\n",
    "    trading_calendar,\n",
    "    assets,\n",
    "    factor_start_date,\n",
    "    universe_end_date)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Format alpha factors and pricing for Alphalens\n",
    "In order to use a lot of the alphalens functions, we need to aligned the indices and convert the time to unix timestamp. In this next cell, we'll do just that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_factor_data = {\n",
    "    factor: al.utils.get_clean_factor_and_forward_returns(factor=factor_data, prices=pricing, periods=[1])\n",
    "    for factor, factor_data in all_factors.iteritems()}\n",
    "\n",
    "unixt_factor_data = {\n",
    "    factor: factor_data.set_index(pd.MultiIndex.from_tuples(\n",
    "        [(x.timestamp(), y) for x, y in factor_data.index.values],\n",
    "        names=['date', 'asset']))\n",
    "    for factor, factor_data in clean_factor_data.items()}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quantile Analysis\n",
    "### Factor Returns\n",
    "Let's view the factor returns over time. We should be seeing it generally move up and to the right."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls_factor_returns = pd.DataFrame()\n",
    "\n",
    "for factor, factor_data in clean_factor_data.items():\n",
    "    ls_factor_returns[factor] = al.performance.factor_returns(factor_data).iloc[:, 0]\n",
    "\n",
    "(1+ls_factor_returns).cumprod().plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basis Points Per Day per Quantile\n",
    "It is not enough to look just at the factor weighted return. A good alpha is also monotonic in quantiles. Let's looks the basis points for the factor returns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "qr_factor_returns = pd.DataFrame()\n",
    "\n",
    "for factor, factor_data in unixt_factor_data.items():\n",
    "    qr_factor_returns[factor] = al.performance.mean_return_by_quantile(factor_data)[0].iloc[:, 0]\n",
    "\n",
    "(10000*qr_factor_returns).plot.bar(\n",
    "    subplots=True,\n",
    "    sharey=True,\n",
    "    layout=(4,2),\n",
    "    figsize=(14, 14),\n",
    "    legend=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you observe?\n",
    "\n",
    "- None of these alphas are **strictly monotonic**; this should lead you to question why this is? Further research and refinement of the alphas needs to be done. What is it about these alphas that leads to the highest ranking stocks in all alphas except MR 5D smoothed to *not* perform the best.\n",
    "- The majority of the return is coming from the **short side** in all these alphas. The negative return in quintile 1 is very large in all alphas. This could also a cause for concern becuase when you short stocks, you need to locate the short; shorts can be expensive or not available at all.\n",
    "- If you look at the magnitude of the return spread (i.e., Q1 minus Q5), we are working with daily returns in the 0.03%, i.e., **3 basis points**, neighborhood *before all transaction costs, shorting costs, etc.*. Assuming 252 days in a year, that's 7.56% return annualized. Transaction costs may cut this in half. As such, it should be clear that these alphas can only survive in an institutional setting and that leverage will likely need to be applied in order to achieve an attractive return.\n",
    "\n",
    "## Turnover Analysis\n",
    "\n",
    "Without doing a full and formal backtest, we can analyze how stable the alphas are over time. Stability in this sense means that from period to period, the alpha ranks do not change much. Since trading is costly, we always prefer, all other things being equal, that the ranks do not change significantly per period. We can measure this with the **factor rank autocorrelation (FRA)**.\n",
    "\n",
    "[alphalens.performance.factor_rank_autocorrelation](https://quantopian.github.io/alphalens/alphalens.html?highlight=factor_rank_autocorrelation#alphalens.performance.factor_rank_autocorrelation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls_FRA = pd.DataFrame()\n",
    "\n",
    "for factor, factor_data in unixt_factor_data.items():\n",
    "    ls_FRA[factor] = al.performance.factor_rank_autocorrelation(factor_data)\n",
    "\n",
    "ls_FRA.plot(title=\"Factor Rank Autocorrelation\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sharpe Ratio of the Alphas\n",
    "\n",
    "The last analysis we'll do on the factors will be sharpe ratio. Implement `sharpe_ratio` to calculate the sharpe ratio of factor returns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sharpe_ratio(factor_returns, annualization_factor):\n",
    "    \"\"\"\n",
    "    Get the sharpe ratio for each factor for the entire period\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    factor_returns : DataFrame\n",
    "        Factor returns for each factor and date\n",
    "    annualization_factor: float\n",
    "        Annualization Factor\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    sharpe_ratio : Pandas Series of floats\n",
    "        Sharpe ratio\n",
    "    \"\"\"\n",
    "    \n",
    "    #TODO: Implement function\n",
    "    \n",
    "    return None\n",
    "\n",
    "\n",
    "project_tests.test_sharpe_ratio(sharpe_ratio)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View Data\n",
    "Let's see what the sharpe ratio for the factors are. Generally, a Sharpe Ratio of near 1.0 or higher is an acceptable single alpha for this universe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "daily_annualization_factor = np.sqrt(252)\n",
    "sharpe_ratio(ls_factor_returns, daily_annualization_factor).round(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question: What do you think would happen if we smooth the momentum factor? Would the performance increase, decrease, or no major change? Why?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_#TODO: Put Answer In this Cell_\n",
    "\n",
    "**Answer:** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Combined Alpha Vector\n",
    "\n",
    "To use these alphas in a portfolio, we need to combine them somehow so we get a single score per stock. This is a area where machine learning can be very helpful. In this module, however, we will take the simplest approach of combination: simply averaging the scores from each alpha."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "selected_factors = all_factors.columns[[1, 2, 4]]\n",
    "print('Selected Factors: {}'.format(', '.join(selected_factors)))\n",
    "\n",
    "all_factors['alpha_vector'] = all_factors[selected_factors].mean(axis=1)\n",
    "alphas = all_factors[['alpha_vector']]\n",
    "alpha_vector = alphas.loc[all_factors.index.get_level_values(0)[-1]]\n",
    "alpha_vector.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optimal Portfolio Constrained by Risk Model\n",
    "You have an alpha model and a risk model. Let's find a portfolio that trades as close as possible to the alpha model but limiting risk as measured by the risk model. You'll be building thie optimizer for this portfolio. To help you out. we have provided you with an abstract class called `AbstractOptimalHoldings`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from abc import ABC, abstractmethod\n",
    "\n",
    "\n",
    "class AbstractOptimalHoldings(ABC):    \n",
    "    @abstractmethod\n",
    "    def _get_obj(self, weights, alpha_vector):\n",
    "        \"\"\"\n",
    "        Get the objective function\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        weights : CVXPY Variable\n",
    "            Portfolio weights\n",
    "        alpha_vector : DataFrame\n",
    "            Alpha vector\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        objective : CVXPY Objective\n",
    "            Objective function\n",
    "        \"\"\"\n",
    "        \n",
    "        raise NotImplementedError()\n",
    "    \n",
    "    @abstractmethod\n",
    "    def _get_constraints(self, weights, factor_betas, risk):\n",
    "        \"\"\"\n",
    "        Get the constraints\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        weights : CVXPY Variable\n",
    "            Portfolio weights\n",
    "        factor_betas : 2 dimensional Ndarray\n",
    "            Factor betas\n",
    "        risk: CVXPY Atom\n",
    "            Predicted variance of the portfolio returns\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        constraints : List of CVXPY Constraint\n",
    "            Constraints\n",
    "        \"\"\"\n",
    "        \n",
    "        raise NotImplementedError()\n",
    "        \n",
    "    def _get_risk(self, weights, factor_betas, alpha_vector_index, factor_cov_matrix, idiosyncratic_var_vector):\n",
    "        f = factor_betas.loc[alpha_vector_index].values.T * weights\n",
    "        X = factor_cov_matrix\n",
    "        S = np.diag(idiosyncratic_var_vector.loc[alpha_vector_index].values.flatten())\n",
    "\n",
    "        return cvx.quad_form(f, X) + cvx.quad_form(weights, S)\n",
    "    \n",
    "    def find(self, alpha_vector, factor_betas, factor_cov_matrix, idiosyncratic_var_vector):\n",
    "        weights = cvx.Variable(len(alpha_vector))\n",
    "        risk = self._get_risk(weights, factor_betas, alpha_vector.index, factor_cov_matrix, idiosyncratic_var_vector)\n",
    "        \n",
    "        obj = self._get_obj(weights, alpha_vector)\n",
    "        constraints = self._get_constraints(weights, factor_betas.loc[alpha_vector.index].values, risk)\n",
    "        \n",
    "        prob = cvx.Problem(obj, constraints)\n",
    "        prob.solve(max_iters=500)\n",
    "\n",
    "        optimal_weights = np.asarray(weights.value).flatten()\n",
    "        \n",
    "        return pd.DataFrame(data=optimal_weights, index=alpha_vector.index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objective and Constraints\n",
    "Using this class as a base class, you'll implement the `OptimalHoldings` class. There's two functions that need to be implemented in this class, the `_get_obj` and `_get_constraints` functions.\n",
    "\n",
    "The `_get_obj` function should return an CVXPY objective function that maximizes $ \\alpha^T * x \\\\ $, where $ x $ is the portfolio weights and $ \\alpha $ is the alpha vector.\n",
    "\n",
    "The `_get_constraints` function should return a list of the following constraints:\n",
    "- $ r \\leq risk_{\\text{cap}}^2 \\\\ $\n",
    "- $ B^T * x \\preceq factor_{\\text{max}} \\\\ $\n",
    "- $ B^T * x \\succeq factor_{\\text{min}} \\\\ $\n",
    "- $ x^T\\mathbb{1} = 0 \\\\ $\n",
    "- $ \\|x\\|_1 \\leq 1 \\\\ $\n",
    "- $ x \\succeq weights_{\\text{min}} \\\\ $\n",
    "- $ x \\preceq weights_{\\text{max}} $\n",
    "\n",
    "Where $ x $ is the portfolio weights, $ B $ is the factor betas, and $ r $ is the portfolio risk\n",
    "\n",
    "The first constraint is that the predicted risk be less than some maximum limit. The second and third constraints are on the maximum and minimum portfolio factor exposures. The fourth constraint is the \"market neutral constraint: the sum of the weights must be zero. The fifth constraint is the leverage constraint: the sum of the absolute value of the weights must be less than or equal to 1.0. The last are some minimum and maximum limits on individual holdings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class OptimalHoldings(AbstractOptimalHoldings):\n",
    "    def _get_obj(self, weights, alpha_vector):\n",
    "        \"\"\"\n",
    "        Get the objective function\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        weights : CVXPY Variable\n",
    "            Portfolio weights\n",
    "        alpha_vector : DataFrame\n",
    "            Alpha vector\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        objective : CVXPY Objective\n",
    "            Objective function\n",
    "        \"\"\"\n",
    "        assert(len(alpha_vector.columns) == 1)\n",
    "\n",
    "        #TODO: Implement function\n",
    "        \n",
    "        return None\n",
    "    \n",
    "    def _get_constraints(self, weights, factor_betas, risk):\n",
    "        \"\"\"\n",
    "        Get the constraints\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        weights : CVXPY Variable\n",
    "            Portfolio weights\n",
    "        factor_betas : 2 dimensional Ndarray\n",
    "            Factor betas\n",
    "        risk: CVXPY Atom\n",
    "            Predicted variance of the portfolio returns\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        constraints : List of CVXPY Constraint\n",
    "            Constraints\n",
    "        \"\"\"\n",
    "        assert(len(factor_betas.shape) == 2)\n",
    "        \n",
    "        #TODO: Implement function\n",
    "        \n",
    "        return None\n",
    "\n",
    "    def __init__(self, risk_cap=0.05, factor_max=10.0, factor_min=-10.0, weights_max=0.55, weights_min=-0.55):\n",
    "        self.risk_cap=risk_cap\n",
    "        self.factor_max=factor_max\n",
    "        self.factor_min=factor_min\n",
    "        self.weights_max=weights_max\n",
    "        self.weights_min=weights_min\n",
    "\n",
    "\n",
    "project_tests.test_optimal_holdings_get_obj(OptimalHoldings)\n",
    "project_tests.test_optimal_holdings_get_constraints(OptimalHoldings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View Data\n",
    "With the `OptimalHoldings` class implemented, let's see the weights it generates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimal_weights = OptimalHoldings().find(alpha_vector, risk_model['factor_betas'], risk_model['factor_cov_matrix'], risk_model['idiosyncratic_var_vector'])\n",
    "\n",
    "optimal_weights.plot.bar(legend=None, title='Portfolio % Holdings by Stock')\n",
    "x_axis = plt.axes().get_xaxis()\n",
    "x_axis.set_visible(False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yikes. It put most of the weight in a few stocks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "project_helper.get_factor_exposures(risk_model['factor_betas'], optimal_weights).plot.bar(\n",
    "    title='Portfolio Net Factor Exposures',\n",
    "    legend=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimize with a Regularization Parameter\n",
    "In order to enforce diversification, we'll use regularization in the objective function. We'll create a new class called `OptimalHoldingsRegualization` which gets its constraints from the `OptimalHoldings` class. In this new class, implement the `_get_obj` function to return a CVXPY objective function that maximize $ \\alpha^T * x + \\lambda\\|x\\|_2\\\\ $, where $ x $ is the portfolio weights, $ \\alpha $ is the alpha vector, and $ \\lambda $ is the regularization parameter.\n",
    "\n",
    "**Note:** * $ \\lambda $ is located in `self.lambda_reg`. *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class OptimalHoldingsRegualization(OptimalHoldings):\n",
    "    def _get_obj(self, weights, alpha_vector):\n",
    "        \"\"\"\n",
    "        Get the objective function\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        weights : CVXPY Variable\n",
    "            Portfolio weights\n",
    "        alpha_vector : DataFrame\n",
    "            Alpha vector\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        objective : CVXPY Objective\n",
    "            Objective function\n",
    "        \"\"\"\n",
    "        assert(len(alpha_vector.columns) == 1)\n",
    "        \n",
    "        #TODO: Implement function\n",
    "        \n",
    "        return None\n",
    "\n",
    "    def __init__(self, lambda_reg=0.5, risk_cap=0.05, factor_max=10.0, factor_min=-10.0, weights_max=0.55, weights_min=-0.55):\n",
    "        self.lambda_reg = lambda_reg\n",
    "        self.risk_cap=risk_cap\n",
    "        self.factor_max=factor_max\n",
    "        self.factor_min=factor_min\n",
    "        self.weights_max=weights_max\n",
    "        self.weights_min=weights_min\n",
    "        \n",
    "\n",
    "project_tests.test_optimal_holdings_regualization_get_obj(OptimalHoldingsRegualization)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimal_weights_1 = OptimalHoldingsRegualization(lambda_reg=5.0).find(alpha_vector, risk_model['factor_betas'], risk_model['factor_cov_matrix'], risk_model['idiosyncratic_var_vector'])\n",
    "\n",
    "optimal_weights_1.plot.bar(legend=None, title='Portfolio % Holdings by Stock')\n",
    "x_axis = plt.axes().get_xaxis()\n",
    "x_axis.set_visible(False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice. Well diversified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "project_helper.get_factor_exposures(risk_model['factor_betas'], optimal_weights_1).plot.bar(\n",
    "    title='Portfolio Net Factor Exposures',\n",
    "    legend=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimize with a Strict Factor Constraints and Target Weighting\n",
    "Another common formulation is to take a predefined target weighting, $x^*$ (e.g., a quantile portfolio), and solve to get as close to that portfolio while respecting portfolio-level constraints. For this next class, `OptimalHoldingsStrictFactor`, you'll implement the `_get_obj` function to minimize on on $ \\|x - x^*\\|_2 $, where $ x $ is the portfolio weights  $ x^* $ is the target weighting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class OptimalHoldingsStrictFactor(OptimalHoldings):\n",
    "    def _get_obj(self, weights, alpha_vector):\n",
    "        \"\"\"\n",
    "        Get the objective function\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        weights : CVXPY Variable\n",
    "            Portfolio weights\n",
    "        alpha_vector : DataFrame\n",
    "            Alpha vector\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        objective : CVXPY Objective\n",
    "            Objective function\n",
    "        \"\"\"\n",
    "        assert(len(alpha_vector.columns) == 1)\n",
    "        \n",
    "        #TODO: Implement function\n",
    "        \n",
    "        return None\n",
    "\n",
    "\n",
    "project_tests.test_optimal_holdings_strict_factor_get_obj(OptimalHoldingsStrictFactor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimal_weights_2 = OptimalHoldingsStrictFactor(\n",
    "    weights_max=0.02,\n",
    "    weights_min=-0.02,\n",
    "    risk_cap=0.0015,\n",
    "    factor_max=0.015,\n",
    "    factor_min=-0.015).find(alpha_vector, risk_model['factor_betas'], risk_model['factor_cov_matrix'], risk_model['idiosyncratic_var_vector'])\n",
    "\n",
    "optimal_weights_2.plot.bar(legend=None, title='Portfolio % Holdings by Stock')\n",
    "x_axis = plt.axes().get_xaxis()\n",
    "x_axis.set_visible(False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "project_helper.get_factor_exposures(risk_model['factor_betas'], optimal_weights_2).plot.bar(\n",
    "    title='Portfolio Net Factor Exposures',\n",
    "    legend=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submission\n",
    "Now that you're done with the project, it's time to submit it. Click the submit button in the bottom right. One of our reviewers will give you feedback on your project with a pass or not passed grade. You can continue to the next section while you wait for feedback."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
